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ABSTRACT 

A modeling strategy for the validation and analysis of large- 
scale optimization models is defined and demonstrated. The strategy 
is based on nine principles of analysis and eight principles of 
visualization that are applied in a user controlled hierarchical 
structure which is customized to a particular optimization problem. 
For each model a set of analytic tools, such as spreadsheets and 
graphs, 1s structured to validate and verify data and analyze the 
model and its results. These tools can be quickly recreated with 
data from subsequent runs of the model and sensitivity analysis 
conducted and comparisons made. As a demonstration the strategy is 
applied to PHOENIX, a large-scale U.S. Army helicopter force 
planning model. The strategy incorporates available technology 
using commercially prepared software and a computer workstation. 
The application of techniques such as hypertext, data access and 
backward compatibility enhance the ease of use and effectiveness of 


this approach. 
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I. INTRODUCTION 


Mathematical programming and optimization are playing an ever- 
increasing role in industry and government. It is believed that 
this trend will continue well into the 21st century. Optimization 
is now being used in medicine and health care, environmental 
management, and securities and finance in addition to the more 
traditional operations research fields of transportation, production 
and energy. Users realize that even small improvements in 
operations can sometimes save millions of dollars. The widespread 
availability of desktop optimizers and powerful computers has 
already and will continue to contribute to this trend.  (Hirshfeld, 
errand oc mez and Pulleyolank, 1991) 

Operations research has made significant contributions in these 
fields by representing a problem as a system of mathematical 
equations and solving for a single optimal solution. Büt as 
advances in scientific computation allow for increasing complexity 
of mathematical models, it becomes apparent that they are more 
difficult to analyze. Many of the models now represent a recurring 
Sltuation or involve a multi-period planning horizon. Such models 


are costly and time-consuming to develop, but this is offset by 


their usefulness as planning tools which can be used over and over 
again. Long-term model use frequently results in second-generation 
users, who are not familiar with the original model development, 
being required to execute and analyze follow-on runs. Even model 
developers may have difficulty retaining the intricacies of the 
problem after significant time lapses between runs. 

Historically, models of multi-faceted problems had to have been 
cut down to size and drastically simplified to bring them within the 
scope of human thinking powers, and computing technology. This is 
no longer a satisfactory method of coping with many of the complex 
dynamic systems being modeled and solved today. Due to the rapid 
advances in the storage and processing capacity of today’s computers 
and the increased prevalence of human understanding and use of these 
mechanical abilities, it is no longer necessary to simplify these 
models. Today's models are extremely elaborate, modeling a 
multitude of intricate situations through the use of thousands of 
linear and non-linear constraints and real, and integer variables. 
Larger and more complex problems can now be modeled and solved 
faster and more efficiently as computers have the capability to 
handle larger data bases, more constraints, more complex 


computations, greater storage, and more detailed displays. 


Just as the models are no longer simple, the decisions 
associated with the problems are now more diverse. Specifying an 
objective function value is often an attempt to minimize or maximize 
some ambiguous measure of effectiveness. Similarly, many of the 
bounds on constraints are subjective and not easily defined as they 
are often measures of some intangible quality such as efficiency, 
durability, or some other performance criteria. The use of elastic 
constraints compensates somewhat for these subjectively assigned 
constraint bounds. It allows the constraints to be violated but 
only at some cost internal to the model. 

Due to the complexity of problems being solved through 
optimization an analyst needs to pursue and explore formulations 
and solutions which most accurately describe the situation and 
provide not only feasible, but practical courses of action. 
Technological advances in the past decade have resulted in a many- 
fold increase in speed and availability of computer computational 
power and significant decreases in the cost to solve large models. 
This has supported the analyst's ability to process data for several 
optimal solutions or a range of feasible solutions, enhancing the 
traditional approach of pursuing a single optimal answer to a 
question or problem. Comparisons of these multiple optimal 


solutions can be made and the effects of changing constraint 


conditions can be fully explored. As a necessity, analysts have 
become far more capable in their abilities to conduct sensitivity 
analysis of the effect of data changes on the solution even when 
thousands of constraints are involved. 

Technology, however, has not yet reached all aspects of the 
modeling process. Large, complex models require voluminous input 
and generate voluminous output. Multiple runs of the model compound 
this situation. The analysis required to obtain the best possible 
decision on the most realistic model possible must continue to 
become more sophisticated. Today's analysts are better able to take 
realistic account of the incomplete information and inconsistency 
of a complex and changing world and of the compromises and 
approximations that must be made to fit real-world problems into the 
quantitative terms of a model. As always, analysis is the key to 
both the model and the solution of large-scale optimization. (Jones, 
1988, p. 891; Linstone, 1985, pp. 77-84; and Simon et al 29s 7 EMEN 
] SES) 

The challenge now is to combine these aspects of modeling into 
a total optimization system which will meet the need for "more 
complete and integrated optimization solutions comprised of model 
preparation facilities, analysis tools including visualization, and 


easy access to enterprise-wide data in addition to the solver 


capable ss -senulezeand ,fulleyvblank, 1991.2 p. 21). As the 
problems become more complex and increasing technology allows faster 
Ssolluctons, ene limiting factor in optimization is the time and 
energy required of the analyst to sort through input and output data 
to find and interpret the solutions. 

A naive view considers optimization to be a forward progression 
Sarough. £our basic stages: problem definitión, data collection, 
model formulation and analysis of the results. Analysts realize 
that there is much more to effectively employing large-scale 
optimization models to solve today’s complex problems. There is a 
widely used, though seldom recognized, interactive optimization 
process in which analysis is conducted at each of these four steps. 
As data is collected, it must be verified and validated. The model 
must be compared with the data to ensure feasibility and both the 
model and its solution must be compared against the problem to 
ensure fidelity. Sensitivity analysis is accomplished by comparing 
puccresults or multiple runs of the model. 

The modeling strategy for interactive optimization presented in 
this thesis seeks to harness available technology to vastly improve 
and speed this analysis. It 1S an approach to the interactive 
optimization process that consists of designing a series of analytic 


tools such as graphs and spreadsheets customized to a particular 


application. These tools can then be used by an analyst on 
Subsequent runs and easily extended as necessary to new information 
and considerations. It is especially helpful in the analysis of 
multi-year and recurrent models and it reduces the information gap 
and learning curve between periodic uses and for second-generation 
and subsequent analysts. The strategy can be applied to a large 
variety of models. It 1S based on a set of analysis and 
visualization principles that are uniquely tailored to a particular 
model and driven by that model and the needs of the analyst. The 
implementation of this modeling strategy will reduce the time and 
effort to validate data and interpret solutions allowing the analyst 
to better pursue model development and analysis to make better 
decisions. 

These analysis and visualization principles are presented in 
Chapter II along with a more detailed discussion of the interactive 
Optimization process. Background on human visual processing and 
computer technology is discussed in Chapter III. Chapter IV is a 
description of the user for which this strategy is developed. 
Chapter V outlines the PHOENIX model, a U.S. Army helicopter force 
planning tool. Chapter VI demonstrates the application of the 
modeling strategy to PHOENIX. Chapter VII describes the future of 


the PHOENIX model. The conclusions are presented in Chapter VIII. 


II. THE MODELING STRATEGY 


A. INTERACTIVE OPTIMIZATION PROCESS 

In order to develop an optimization modeling strategy an in- 
depth understanding of the interactive optimization process is 
required. As conceptualized in Figure 1, the four basic parts of 
the process are problem definition, data collection, model 
formulation and analysis of results. The main relationships among 
these parts are depicted by the forward arrows. The backward arrows 
represent the relationships that are more subtle, yet crucial, in 
the optimization process. 

As data is collected it must be consolidated, verified and 
validated. Visual inspection of the raw data files or printouts is 
Simply not feasible. The input data often originates at several 
sources and needs not only to be verified for accuracy, but also to 
be compared for consistency among sources. Without correct input 
data, the results of the model are worthless. The verification and 
validation of input data needs to be an integral part of 
optimization. It must be done both as the data stands alone and as 


it is linked together with the model. 


All aspects of the problem that are critical to the solution 
must be enumerated and accounted for in mathematical formulas. 
Modeling requires that an objective function and constraints, often 
qualitative terms, be quantified in some way. This presents a 
difficulty in that real world situations are complex dynamic systems 
which can seldom be translated into black and white values and it 
is often impossible to establish a hierarchy of these values. If 
there are mutually exclusive requirements, elastic constraints are 
introduced to ensure that the unfulfilled requirement is at some 
cost internal to the model. 

Both the model and its solution must be checked against the 
actual problem to ensure fidelity, validity and practicality. If 
not adequately representative of the situation being modeled, 
changes may need to be made in the data collection, model 
formulation, or even problem definition. The analyst must often 
explore alternate solutions obtained by varying the input parameters 
and constraint conditions to reflect the dynamics of a complex 
problem. Even small changes made to enhance the application of the 
model can indicate an entirely new course of action. 

In order to conduct sensitivity analysis of large-scale 
problems, the results of multiple runs must be compared. The 


voluminous data produced by each run requires the same in-depth 


analysis as the original output. Thorough comparisons between 
successive runs highlight the advantages of one over another and aid 
in the selection of one optimal or a range of feasible solutions. 

Faster more efficient computers allow the solution of these 
large-scale problems. Their final results can be displayed and 
presented very effectively using a variety of sophisticated 
presentation tools. Technology, however, has not yet reached the 
relationships in the interactive optimization process represented 
by the back arrows. There will not be a general way to bring 
technology to these relationships since each model is an individual 
problem with a hierarchy of analysis consisting of unique questions 
and decisions to be made about the optimization output. The 
modeling strategy developed in this thesis can help to bring 
technology to the interactive optimization process by defining a set 
of analysis and visualization principles that can be applied in 


different combinations to all models. 


B. PRINCIPLES OF THE MODELING STRATEGY 

The analysis and visualization principles that are the basis of 
the modeling strategy will be briefly explained in this section. 
Although it would be possible to more fully develop each of these 
principles, a more effective way to present the strategy is to apply 


it to a real, contemporary, complex, and important model. The 


Strategy will be applied to the PHOENIX model in Chapter V. Each 
principle will be demonstrated, however the focus is not on 
individual principles, but on the combinations, interactions and 
compromises among these principles as they are applied to an actual 


model. 
d Analysis Principles 


e (Al) HIERARCHICAL STRUCTURE. Analysis progresses from a broad 
overview of the objective function value down to a single piece 
of information. The specific hierarchy will be designed by the 
analyst and will depend on the nature of the problem. Levels 
may be skipped and crossovers between branches can occur to 
meet the needs of the analyst. 


e (A2) USER CONTROL. The analyst is able to navigate the 
hierarchical structure to answer the questions that arise as 
part of a specific analysis. 


* (A3) DATA ACCESS. At all points in the analysis process, the 
analyst must have easy access to specific data values and the 
relationships among those values. 


* (A4) MULTIPLE REPRESENTATIONS. Different aspects of the same 
data are extremely valuable in revealing information about the 
relationships among data. These varied representations are 
easily constructed and readily available. 


e (A5) SIDE-BY-SIDE COMPARISONS. Side-by-side comparisons of 
similar information enhance the analysis process by decreasing 
the amount of decoding that is required and by highlighting 
differences. 


* (A6) BASE CASE. In model development and sensitivity analysis, 
a base case that is believed to best represent the problem can 
be used. Results of other runs are compared against the base 
case. 


TO 


(A7) CUSTOM MODEL. The nature of the problem and the questions 
inherent to it drive the set of representations that are used 
for analysis of that model. 


(A8) EXTENSIBLE. Changes made to a particular representation or 
piece of information filter through all representations that 
contain that information. 


(A9) BACKWARD COMPATIBLE. Any additional representations or 
changes to existing representations made during analysis of a 
run are automatically added to any previous runs of the model. 


2: Visualization Principles 


(V1) REPRESENTATION DRIVEN GRAPHS. The results of time-series 
models are most often displayed with the time periods on the 
Þorizontal or x axis. This fits in with the concept otf time 
moving forward and the general perception of the forward 
direction as the one from left to right. Alternately, a model 
which seeks to find the best combination of ingredients in 
order to make some new product may be best represented by 
filled bar charts where the height of the bar represents the 
new product. Pie charts could also be used but are generally 
not considered good analysis tools due to the difficulties 
encountered in perceiving differences in angles and slopes 
(Tufte, 1983, p. 178 and Cleveland and McGill, 1985, p. 829). 
Stacked bar charts are also very effective in showing different 
combinations of things that make a whole. 


IV)" ZOOMING. The dbrlTty- to focus or, 200m 11 (On areas otf 
interest or out to the big picture is crucial in the analysis 
process. The focus could be on certain time periods, on 
information pertaining to a particular constraint, or even on 
the objective function value. It allows the analyst to pursue 
a separate train of thought or to redirect analysis in a 
Certain direction. 


(V3) DATA ACCESS. The analyst must also have simple, easy 
access to the data behind the graphs. Whereas the graph 
provides a way to view the information in relation to other 
information in the model, the actual data is necessary for 
quantitative analysis. 
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(V4) APPROPRIATE DISPLAYS. Not all information is most 
effectively displayed in graphic form. The analyst must decide 
when a graph is needed and when displaying the data in tabular 
form 1S more informative. 


(V5) SIMPLICITY. Visualization tools must be designed to reveal 
information rather than draw attention to the display. They 
Should avoid distortion of the data and allow the analyst to 
interpret it.: (Tufte mle TEPA 


(v6) CONSISTENCY. Graphs that are consistent in size, coloring 
and labeling with other graphs which display similar 
information reduce the amount OE time required fob 
interpretation. 


(V7) MULTIPLES. An especially effective method to display and 
highlight both subtle and extreme differences is the use of 
small multiples which are a series of graphics, each showing 
the same combination of variables, indexed by changes in 
another variable. Since the design stays constant, the 
analyst's attention can be devoted entirely to the changes in 
the data (Tufte, 1983, P e170); 


ali? 


III. BACKGROUND 


A. HUMAN VISUAL INFORMATION PROCESSING 
Extensive research has been conducted on human visual 

perception and information processing. The physiology has been 
explored (Marr, 1982, pp. 4-15) and numerous experiments conducted 
(Powers et al, 1983 and DeSanctis, 1984) to determine the benefits 
of using representation (graphics and tables) versus description 
(text) to enhance the human's ability to process, recall, and 
analyze visual information. Under a variety of circumstances and 
experimental conditions, some studies conclude the superiority of 
graphical presentation over text or tables and some determine that 
no one method can conclusively be determined superior. When 
evaluated, Powers et al determined that the combination of graphical 
and tabular data was more effective than either method used alone 
MUS. D. 558). Dependent variables evaluated in these studies 
differed, but a typical list comes from DeSanctis (1984, p. 468): 

+ Interpretation accuracy; 

e Problem comprehension; 

e Task performance; 


e Decision quality; 
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e Speed of comprehension; 

e Decision speed; 

e Memory for information (recognition and recall); 

e Viewer preference. 

Further research more specifically explores effective 
techniques for presenting quantitative information. In his books, 
Edward Tufte asserts that "well-designed data mecnm are usually 
the simplest and ... most powerful of all methods for analyzing and 
communicating statistical information" and that the general 
principles of effective design are universal, not tied to language 
or culture (1983; pp. G@=9 andis UE Ole "Often the most 
effective way to describe, explore, and summarize a set of numbers - 

even a very large set - is to look at pictures of those numbers." 
(Tutte, I983 pb. 9) To communicate complex ideas with clarity, 
precision, and efficiency, Tufte lists the important elements of 
graphical displays (1983, p. 23): 

e Show the data; 

e Induce the viewer to think about the substance rather than 
about methodology, graphic design, the technology of graphic 
production, or something else; 

e Avoid distorting what the data have to say; 

e Present many numbers in a small space; 


e Make large data sets coherent; 
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e Encourage the eye to compare different pieces of data; 


» Reveal the data at several levels of detail, from a broad 
overview to the fine structure; 


e Serve a reasonably clear purpose: description, exploration, 
tabulation, or decoration; 


e Be closely integrated with the statistical and verbal 
Gescriptions of a data set. 


Cleveland and McGill (1985, p. 828) caution against graphs that are 

too elaborate or technologically sophisticated. They have studied 

the link between graphs and the human visual system and conclude: 
When a graph is constructed, quantitative and categorical 
information is encoded, chiefly through position, shape, size, 
Symbols, and color. When a person looks at a graph, the 
information is visually decoded by the person's visual system. 


A graphical method is successful only if the decoding is 
effective. 


Norbert Enrick (1972, p. 2) cites the values of "well prepared 
charts and graphs" as creating interest, clearly portraying 
relationships, saving time, saving space, providing a synoptic 
overview, unearthing hidden factors, and enhancing thought 
processes, analysis, ideation and creativity. 

In the field of operations research, the primary applications 
of graphic techniques to date have been in the areas of simulation 
and data analysis. While both are applicable to optimization, 
surprisingly little has been done to incorporate graphical methods 


into the analysis process for large-scale optimization models. But 
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as more complex and increasingly larger optimization models and more 
sophisticated operations research techniques are developed 
"understanding the behavior of the underlying system, detecting 
trends, debugging and validating the model become more challenging." 
(Jones, 1988, p.6) Graphical and visual techniques and user 
interaction in the modeling process can be employed to assist 
Significantly in optimization analysis. Incorporating these 
techniques more prominently into operations research has been cited 
as a need in the current growth of the field of operations research, 
and their potential benefits are inestimable. (Jones, 1988, p.7; 


Bell, 1985, p.31; HWPrion, 1920%D 26) 


B. REPRESENTATION/ VISUALIZATION 

With the increasing volume of input data and results that can 
now be produced by large-scale optimization, the problem analysts 
face has changed from making limited calculations to being able to 
understand and interpret the masses of data produced. Effective 
representation is a key factor in the abilities of the analyst to 
validate and analyze this information. Text and tables have long 
been used by optimizers for conveying the data and results of their 
algorithms. But perhaps the most effective and universally 
understood means of representation is through the use of graphics. 


Computer graphics programs that are used to help scientists 
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visualize and thereby better understand their research problems are 
Categorized as scientific visualization (Rivenbark, 1989 ). Along 
the lines of the old adage "a picture is worth a thousand words", 
these visual images can convey more information and reveal aspects 
and relationships not as easily discerned by analyzing formulas or 
perusing numerical values. "Using computer generated images and 
human vision in scientific visualization ... (can) convey a 
tremendous amount of information in a short period of time." 
(Nielson, 1989, p. 10). The strength of visualization is its use 
of the greatest processor available: the human brain, according to 


Lloyd Treinish, a computer scientist with NASA’s National Space 


Science Data Center (NSSDC), because it "takes advantage of the 
inherent power of the human visual system." (Rivenbark, 1989, p. 
36) 


Craig Mundi, Research and Development vice president for 
Alliant Computer Corporation, Littleton, Massachusetts, divides 
visualization into two types: statistically produced data sheets 
from which the user, or computer, extracts a static graphical 
representation, and models that the user dynamically manipulates in 
real time (Jones, 1988). This real-time animation is becoming more 
frequently utilized in the field of operations research for 


animation of algorithms and simulations models. Animation allows 
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the analyst to interact with and thereby redirect or change the 
actual problem during its solution. The solution or the algo esemi 
can also be animated showing the user the actual process behind the 
model and the steps involved in solving it. Currently, however, 
technology cannot accommodate the animation of large-scale 
optimizations with their multitude of variables and constraints. 
Three-dimensional computing has improved Operations Research 
visualizations greatly, but scientific computing has yet to master 
graphing the thousand-dimensional and the human mind would have 
difficulty comprehending it. 

Optimization does not now allow run-time interaction to 
redirect the model or make changes during computation. The reason 
for this is found in the Simplex procedure involved in linear 
programming. The assortment of variables and constraints that make 
up the optimization algorithm form, in effect, a multi-dimensional 
polytope whose extreme points bound the feasible solutions. The 
Simplex optimization process searches these points for the optimal 
solution. Whether using the traditional Simplex algorithm which 
progresses from one adjacent vertex to another until there is no 
improved point, or the more recently developed Karmarkar algorithm 
(Hamilton and Stein, 1989, p.36) which employs a shortcut to work 


through the center of the polytope instead of on the surface, any 
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run-time interaction would change the composition of the polytope 
and invalidate the systematic optimization process. The use of 
Static graphical representations of visualization are more 
appropriate, then, to improving the portrayal of large-scale 
optimizations. 

What visualization can provide to large-scale optimization is 
enhancement and clarity of voluminous input and results. It can be 
used as a vehicle in focusing and directing analysis. The strategy 
described here employs these static visualizations in an interactive 
environment where analysis and changes can be incorporated between 
successive runs of the optimizer to explore alternate decision 
Strategies. The development of graphics programmed directly into 
the output production excuses the analyst from having to become a 
computer programmer and from performing labor intensive data 
manipulations. It speeds the preparation of graphic representations 
of the model and eliminates the need to “reinvent the wheel" as the 
analyst elects to display different formats, change input values and 
parameters, and validate and make comparisons on multiple runs of 
the optimization (Rivenbark, 1989). 

These changes serve to produce better problem solving in the 
field of operations research and enhance the credibility of analysts 


in their work with managers and decision makers who prefer a more 
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socio-technical approach to problem solving. To fully realize the 
potential of mathematical optimization, the effectiveness of 
algorithms and their implementation must be developed to solve the 
actual problems of the user (Schultz and Pulleyblank, 1991). By 
capitalizing on the speed and power of delivery systems in the 
development of a total optimization system, the modeling strategy 


will greatly enhance the capabilities and latitude of the analyst. 


C: COMPUTER TECHNOLOGY 

The great strides made in computer technology have resulted in 
a decline in price coupled with a rise in capability. This puts 
extremely powerful machines within easy reach of analysts and 
researchers. These machines can solve exceptionally large, complex 
problems in relatively short periods of time as they are adequately 
equipped with memory to store the tremendous amount of data and the 
intricate relationships among its elements that these problems 
demand. Along with this increased power come fast enhanced graphics 
that can provide the user with high quality visualization products 
for both analysis and presentation. This is especially important 
for operations research, and even more so for optimization, as it 
allows the modeling, solving and analysis of production sized 
problems with a degree of fidelity that has never before been 


possible. Gregory M. Nielson, in an article in Computer magazine 
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(1989, p.10), captures the advances of computer technology and 

hyghsghtsstbe direction for the future: 
Advances in scientific computation allow increasing complexity 
of mathematical models and simulations. This results in a 
closer approximation to reality, which enhances the possibility 
of acquiring new knowledge and understanding ... The problem is 
Poco ves. all of hrs information ro the ‘scientist to 
effectively use human creativity and analytic capabilities. 

Of the newly affordable computer systems, the workstation may 
have the most profound effect on the operations research analyst. 
In addition to having the power and speed of the latest technology, 
it has several other characteristics which enhance the capabilities 
of an analyst. One advantage is the high speed architecture which 
allows quick transfers among many different environments such as the 
model, the spreadsheet and the graphics. Additionally, most 
workstations employ a windowing system which provides a set of 
programming tools and commands for building the menus, windows, and 
dialogue boxes that appear on a screen. These two features 
alleviate the difficulties and inconveniences of switching computer 
environments and facilitate the simultaneous use of multiple 
applications for comparison and consolidation. 

While workstations are self-contained computers, they are also 
an integral part of the networking concept. In a network, 


information is shared among all the systems which are linked to each 


other via this network. In addition to sharing information, 
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networks can also share the work required to solve large-scale 
problems. One computer is in primary control of a problem but can 
distribute solving tasks to other computers, including other 
workstations,  micro-computers and even a mainframe. The 
distribution is based on both the capability and availability of the 
machines on the network. When a task is completed, its solution or 
status report is sent back to the controlling computer for 
consolidation with the rest of the problem until the entire job is 
complete. This process maximizes the effective use of the computers 
and minimizes the time required to solve a problem. The 
interactive optimization process discussed in this thesis does not 
directly rely on networked problem solution, but there is potential 
for links with other systems to take full advantage of a network 


during the course of an optimization. 
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IV. DESCRIPTION OF TARGET USER 


Currently, off-the-shelf software exists to bring the methods 
of optimization to the general user. Several packages, such as VINO 
and Lotus 1-2-3 utilize spreadsheets and, with simple instructions 
for input of objective function and constraint values, will produce 
an optimal solution without the requirement of user comprehension 
SE optimization principles. Perhaps one of the most capable 
products for simple linear optimization is What's Best which 
integrates the flexibility of the Lotus 1-2-3 spreadsheet with the 
power of LINDO (Bodily, 1986, pp. 41-42). These programs, while 
useful for more simple tasks, are not sophisticated enough for the 
complex problems encountered by operations research analysts. 

The process developed in this thesis will address optimization 
at this higher level. It is designed for a sophisticated user 
trained in operations research and analysis. It requires an in- 
depth understanding of the theory and mechanics of mathematical 
programming, including the principles of sensitivity analysis. A 
working knowledge of data analysis techniques with emphasis on 
graphical representations is also essential. This system is not 


intended for a casual user. A level of involvement is expected that 
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would justify the construction of a project-specific application of 
the process developed. It 1s envisioned that the analyst will have 
a long-standing relationship with a particular project and with 
optimization projects in general. This will ensure that the process 
is used to its fullest extent and that the user will derive the 
maximum benefit from it. Additionally, many long-term projects are 
designed for use as recurring decision support models. The 
techniques, therefore, must be sufficiently simple and generic to 
be successfully transferred among qualified users. 
As a minimum it is expected that the user will have: 


e 6 semester hours of graduate credit in linear programming and 
integer programming methods. 


* 3 semester hours of graduate credit in data analysis 
techniques. 


e familiarity with spreadsheets and their related graphics 
packages. 


* experience with a computer workstation environment. 
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V. THE PHOENIX MODEL 


A. PURPOSE 

PHOENIX 1S currently used by the U.S. Army Concepts Analysis 
Agency (CAA) as a decision aid for helicopter force planning. It 
was developed in late 1987/early 1988 in response to the realization 
by the U.S. Army that it had no comprehensive plan for modernizing 
its helicopter fleet. The helicopter fleet was composed of mostly 
Vietnam-era aircraft which were nearing the end of their useful 
lives. The Army Aviation Modernization Trade-off Requirement 
(AAMTOR) study (Force Systems Directorate, 1988) was commissioned 
to develop a comprehensive force planning decision aid and the 
PHOENIX model was the result. As described by Brown et al (1991), 
PHOENIX "captured complex procurement and modernization tasks in an 
optimization-based decision support system ... which recognizes 
yearly operating, maintenance, retirement, service-life extension, 
and new procurement costs while enforcing constraints on fleet age, 
technology mix, composition and budgets over a multi-year planning 
horizon". The final report of the AAMTOR study describes the model 


development, data collection and analysis in full detail. 
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B. DESCRIPTION 

The portions of the model which are critical "COMM 
understanding of this thesis are summarized in this section and the 
following one. (Force Systems Directorate, 1988). PHOENIX is a 
mixed integer linear program (MILP) variant of a classic operations 
research optimization problem, the equipment replacement model. 
Since the real scenario is more complex than ae classic problem, 
the PHOENIX model is also more elaborate (p. 3-1). PHOENIX 
addresses such concerns as multiple missions of the Army aviation 
fleet, budgetary limits, fixed costs associated with production and 
multiple criteria for mission fulfillment that are not accounted for 
in the basic equipment replacement model. The PHOENIX model was 
solved using a commercial quality optimization package, the "X- 
system" (p. 3-10). The most difficult scenario that was solved was 
over a 25 year planning horizon. It contained 288 binary-valued 
decision variables, 9579 continuous decision variables and 3737 


constraints (DP. 3-10 


Gr GOALS AND ASSUMPTIONS 
The objective of the PHOENIX model is to minimize the sum of 
the operations and maintenance annual expenditures and the penalties 


associated with constraint violations. The annual budget and 
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mission fulfillment requirements are included in the constraints 


of the model. The model determines (p. 5.1): 


Some 


When (time period) aircraft production lines begin and end 
MProaqueceron, Or wt they do at all: 


How many aircraft are purchased from each production line in 
each time period. 


How many operational aircraft in a cohort are improved through 
a Service Life Extension Program (SLEP) in each time period. 


How many operational aircraft in a cohort are retired in each 
time period. 


of the key assumptions of the model are (p.1-8): 


Aircraft are purchased and supported as cohorts composed of all 
aircraft of the same model produced during the same time period 
(year). 


Aircraft age can be managed in years (age of the airframe) vice 
its actual flying hours. 


Aircraft are paid for in budget year t and delivered in budget 
year t + L, where L is the lag time for production. 


All expenditures are planned for in constant dollars. 


Monies not committed in budget year t are not carried forward 
to subsequent years. 


Annual requirements for aircraft include float, training and 
operational needs. 


Fixed costs associated with opening, maintaining and closing an 
alrcraft production line are significant and must be included 
in long-range plans. 


Only one production line may exist for a particular type and 
model of aircraft in any time period. Certain models are 
predecessors to others on an individual production line and 
can't be produced concurrently, while production lines 
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producing different aircraft may operate simultaneously in a 
time period. 


Production lines have minimum and maximum sustaining rates of 
production. 

The user of the model specifies different requirement, 
resource, and policy parameters which are included in the 
constraints of the model and which can influence its solution (p. 
5.1). The requirement parameters are: 

e The number of time periods to consider. 


« The minimum and maximum number of aircraft necessary to satisfy 
each mission in each time period. 


The policy parameters are: 


« The maximum useful life (time periods) of each aircraft in a 
mission fleet in the model. 


e The minimum fraction of each mission fleet to be composed of 
high-technology aircraft in each time period. 


* The maximum average age of each aircraft in a mission fleet in 
each time period. 


The resource parameters are: 

e The minimum and maximum budget available to spend on aircraft 
procurement, aircraft operations and maintenance, and on 
aircraft retirement in each time period. 

e The existing inventory (including year of manufacture and 
number of aircraft) and their technology (high or low) and cost 


characteristics. 


+ The technology and cost characteristics of new aircraft designs 
and design improvements. 
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mothe. preauce wom Wane characteristics, including fixed costs, 
productron Capacities and limits, types or aircraft which can 
be produced and the constraints in opening and closing dates. 

D. IMPORTANCE OF MODELING STRATEGY TO ANALYSIS OF PHOENIX 

PHOENIX is a real, complex, specific, important, contemporary, 
subjective model. Decisions that are made based on its results can 
influence the spending of billions of dollars. Brown et al. (1991) 
discuss some of the difficulties encountered during the modeling and 
analysis of this problem. Many are problems that would have been 
alleviated by use of the modeling strategy described here. In 
particular, they discuss the complications faced in both gathering 
and validating data from multiple sources and in uniting this data 
with the actual model formulation. 

The selection of a tangible and realistic objective function 
was difficult in light of the many different criteria for mission 
success and budget limitations. They also had the difficult task 
of developing a penalty system that accurately reflected the 
relative importance of each of the measures of mission 
accomplishment incorporated into the model as elastic constraints. 
The assignment of many constraints was subjective with the analysts 
using "corporate wisdom ...[to] characterize the current fleet 
status, costs, and likely consequences of future procurement and 


manufacturing options" (Brown et al, 1991). Since this was a 
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complex problem that had never been modeled before and because its 
impact would be so far-reaching, there was great concern about 
ensuring all relevant parameters were accounted for and that the 
model was a true depiction of the problem. The situation was 
further compounded because the model developers were working on 
separate parts of the model on opposite sides of the country. 

The scope of the PHOENIX decision is broad, influencing not 
only the multi-year planning of Army helicopter procurement but 
interacting with the modernization of other Army and Department of 
Defense organizations. The helicopter program is competing for a 
portion of a fixed budget. PHOENIX is a high visibility model that 
must be justified at all levels, from user to congressional, and 
presented to a variety of audiences from technicians and analysts 
to generalists in Congress. 

Because of its importance and the dollars involved, it is 
critical that the analyst explore a large variety of alternate 
solutions obtained by changing the resource, policy and/or 
requirement parameters. It is necessary to fully analyze each of 
the plans since one alternative could provide a lower objective 
function value than another plan, but not be considered a better 
plan due to the constraints that were violated and the corresponding 


money assessed in penalties. A keen understanding of the 
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implications involved in each of the alternate plans is critical to 
the solution of the problem. 

Sixteen instances of the model were run as part of the initial 
Study and two were presented in the report. Results of each 
scenario had to be put into tables, then converted into graphs. 
The goal of the strategy presented here is to use the principles 
of analysis and visualization to augment the validation and analysis 
process. More runs could be analyzed with more conditions 
considered. This would yield greater confidence in the decision 
recommended and help to prepare the results for presentation to 


eeir various users: 


A 


VI. IMPLEMENTATION OF THE STRATEGY 


A. TOOLS 

The analysis and visualization principles presented in Chapter 
II are implemented through a series of spreadsheets and the 
associated graphics. This particular aop Nin is built DS NES 
IMPROV spreadsheets and Presentation Builder graphics on the NEXT 
computer, a low cost ($5000) workstation. Other visualization 
software could be used subject to user preference and availability. 
The specific software packages are not important, rather the 
flexibility and enhanced capabilities that they render allow more 
time to be devoted to analysis rather than to data manipulation. 
This is especially important given a recurring model such as PHOENIX 
which acts as a decision aid in a continuously updated planning 
process. 

The analyst can customize a series of worksheets and graphs 
that is generated each time the optimization is run. It is 
envisioned that each optimization model eS dictate a set of 
spreadsheets and graphs that are uniquely suited to the analysis of 
that project. These tools become part of the optimization process 


and are available to be used and amended as necessary during the 
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analysis. This streamlines the analysis process when multiple runs 
of the model are conducted. The same information for each run is 
in the formatted set of worksheets and graphics, facilitating the 
analysis process and the comparisons between succesSive runs. 

The use of spreadsheets and their associated graphics as tools 
in the optimization process provides significant advances in the 
flexibility of the analysis. The data is easily manipulated via 
simple formulas opening possibilities for recognizing new 
information or new aspects of information that can be derived from 
the raw input data and the results of the model. The data can be 
combined in many different ways or separated into its individual 
pieces as the analyst sees fit. In this way, data is manipulated 
to reveal information. 

Both the worksheets and the graphic packages are three 
dimensional. Additionally, any changes to the data in a spreadsheet 
ripple through all associated spreadsheets and graphics enabling the 
analyst to see how the change effects the other aspects of the 
model. This can be used as a stepping stone in model validation and 
exploration ina limited "What if?" scenario. The results of the 
changes may indicate to the analyst what changes to make to input 


datar Or Constraints- 
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Whereas the IMPROV spreadsheets and Presentation Builder 
graphics are exceptionally capable, they do not as yet incorporate 
all the technology that is currently available in similar software 
packages. Since this technology will be available, it is included 
in the application presented in the next section. In particular, 
hypertext, a software system that Supports special links within a 
single window and between pairs of windows, is applied. Hypertext 
allows a user to enter a software package and program changes to it 
that enhance the capabilities of the software and tailor it to the 
user's needs. Here the term hypertext will refer to the capability 
for the user to zoom in and view only selected information in a 
window or easily move from viewing one window to viewing another, 
usually by mouse clicking on a special button. Additional aspects 
to hypertext systems are discussed in Conklin (1987). 

An application of the analysis and visualization principles to 
a sample run of PHOENIX with a 20-year planning horizon and a 2$ 
real budget growth rate demonstrates the modeling strategy. The 
thought processes and decisions an analyst might make as part of the 
interactive optimization process conceptualized in Figure 1 are 
described. The caption under each graph includes the principles 


that the graph most remarkably illustrates. Some of the principles, 
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such as a customized model and data access, apply to all the graphs 


and are not specifically noted in each individual graph. 


B. VALIDATION AND VERIFICATION 

Of initial importance to the analyst is the assurance of the 
validity, accuracy and consolidation of the input data. The 
Spreadsheet environment is one of the most organized and effective 
means to consolidate and compare data. Data fields can be directly 
imported into a spreadsheet. Other programs, most importantly in 
this case, the optimizer, can directly read from these spreadsheets. 
The spreadsheets are, in turn, directly linked to graphics 
presentations where changes made to data in the spreadsheet are 
automatically communicated to the graphics. 

Using the appropriate graphs, the analyst can look for trends 
and discrepancies from trends as well as outlier values in the data. 
It is generally obvious from the nature of the data what trends 
should be exhibited and any deviation from these trends should be 
examined. For example, budgets over time tend to increase or remain 
constant, so any short term decrease would be a signal to the 
analyst for further exploration. DPtoocoüld be'gnrecative 'or-un 
actual trend or it may simply be the result of incorrect data. The 
labels in the spreadsheets facilitate the verification of isolated 


values and sparse elements of information. This process is 
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ultimately more effective and less time-consuming than manual 
scrutiny of raw data files. The input data for the PHOENIX model 
consists of both isolated values and sparse elements; those that 
will display a trend and those that are unrelated pieces of 
information that need to be individually scrutinized. For example 
the budget limits are easily verified for consistency in Figure 2. 

It is important to realize that all dollar figures in PHOENIX 
have been adjusted for inflation and are displayed in 1988 dollars. 
Thus, it is the assumption in this run of the model that there will 
be an increase in the budget maximum limit over the time horizon of 
the model where the lower limit will remain constant after 1995. 

Associated with the budget are several growth rates that can 
affect the problem solution. They are the budget growth rate, the 
inflation rate and the O&M growth rate displayed in Figure 3. In 
addition to the actual values, it is interesting to see these 
magnitudes in relation to each other and to verify that this is 
consistent with the trends in current government spending. 

Three other areas which should display trends are in the 
mission capability areas of maximum age, high-technology fraction 
and force requirements, shown in Figure 4. Since the goal of 
PHOENIX is to ensure that the Army helicopters are sufficiently 


capable of performing their mission in the future, it is logical to 
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assume that the trends should be for helicopters that are better 
equipped technologically with a lower maximum age. These 
accomplishments would allow for a reduction in current force size 
to some constant level. See Figure 4. Any deviations from these 
trends, such as the slight dip in high-technology fraction goals in 
1999, are apparent to the analyst who will then attempt to discover 
their cause. Line graphs such as these vastly reduce the amount of 
time devoted to data validation and become extremely useful in 
analysis and presentation of results of the model. 

When graphically displayed, time series data such as the 
scheduling information shown in Figure 5 becomes much more 
informative. Recalling that PHOENIX demands that only one 
production line be open at a time for each aircraft mission and that 
certain lines are predecessors to others, this graph 1S a ready 
reference for determining when each production line may open and 
close. 

Some other information which requires verification is not as 
meaningful when depicted graphically since it shows no trends or 
consistencies. In PHOENIX, this data includes information on 
alrcraft such as purchasing cost, last high-technology year and the 
other information shown in Figure 6. The spreadsheet format greatly 


enhances the ability of the analyst to both understand the 
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information and verify its accuracy. The analyst also has access 
to other information in spreadsheet format such as production line 
capacities and initial force Compositions. Simple spreadsheet 
formulas can easily be used to compare and consolidate information. 

After this validation and consolidation is complete and any 
necessary corrections made, the optimization model is ready to run. 
Spreadsheets and graphs are again utilized to display the results 
as part of the interactive process. The particular views that are 
used here have been selected as part of the PHOENIX optimization 
process but can easily be manipulated to highlight any information 


Whlchem c oM e resis 


C. SINGLE-RUN ANALYSIS 

Once the optimization is run, the analyst begins the analysis 
of overall results. The progression through the analytical 
hierarchy may vary by analyst and situation and the visualization 
tools can be tailored to accommodate this. Of initial interest is 
the objective function value. Recall that in PHOENIX the objective 
function is to minimize the sum of the O&M costs and the penalties. 
Look at both the total objective function value and its components 
in Figure 7. Due to the multi-year planning horizon of PHOENIX, the 
analyst may gain more insight from examining the annual components 


of the objective function value in Figure 8. At this point, the 
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analyst could choose to proceed in one of the two directions 
depicted in the diagram in Figure 9. 

In an attempt to determine whether the model is accurate and 
provides the "best" solution, the analyst must assess whether 
penalty values are realistic as assigned and if penalties taken 
wrphrin the model are appropriate in finding an optimal solution. 
If the penalties are a large portion of the objective function 
value, further scrutiny of penalties is indicated. If the penalties 
are a small portion of the total or if their relative weight cannot 
be determined at this point, it may be more enlightening to examine 
the total budget and force structure resulting from the optimization 
ERUIT 

From Figure 8, the analysts sees that the O&M values, which 
reflect the real-world costs of Operations and Maintenance, remain 
relatively steady with a gradual increase towards the out-years of 
the run. The penalty values are high at the beginning of the model 
as might be expected because the model decisions have little impact 
on fleet condition in the earliest years. The penalties go to zero 
in the mid-years then become somewhat significant in the second half 
of the planning horizon. If the penalties were large, this would 
invite further analysis but because of their relatively low values 


the analyst may choose to first explore the budget branch. 
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The analyst begins with a look at the total budget expenditures 
of the run in relation to the minimum and maximum budget constraints 
in Figure 10. Recall that these constraints are elastic and incur 
penalties when violated. From Figure 10 the budget exceeded the 
maximum in the first two years then stayed within budget limits for 
the remainder of the run, dropping quickly in the out-years. A 
closer look at expenditures is therefore warranted. 

Although penalties are important to the solution of the model, 
the actual annual expenditures are composed of only the procurement 
and O&M costs. Figure 11 reveals these actual expenditure figures 
along with minimum and maximum budget limits. The last few years 
of a model, in this case 2007 and 2008, may not be valid. This is 
a well-known aspect of such multiple-time period, finite horizon 
models with fixed ending and beginning conditions. It is made much 
more obvious by the visualization techniques used in this analysis. 
This anomaly is caused by the optimizer’s approach to minimizing 
costs and incurring penalties towards the model’s completion rather 
than investing in expensive procurement actions whose long-range 
pay-off would not be realized within the scope of the model. This 
tendency must be considered in viewing and implementing the results 
of the model. Closer examination of Figure 11 indicates a leveling 


off of procurement from the year 2000 and beyond even though annual 
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expenditures are further and further below the increasing maximum 
budget. The analyst would explore the various explanations for this 
trend by asking questions such as "Is the model capable of meeting 
all goals with less than total expenditure?" and "Is the full 
budgeted amount not required?". 

In order to answer these questions, the analyst must first 
determine the answers to more basic questions suchas "What did we 
der?” or "What did we not. get?”. The first question could be 
answered by looking at force compositions, supplemented by age and 
hi-tech compositions. In this instance, however, the question of 
what goals were not met by the solution is more instructive. It is 
best answered by examination of the penalty data. 

Switching "focus EO “the penalty branch Of the analytical 
hierarchy in Figure 9, the analyst would see which goals are not 
being met. Figure 12 shows the amount and type of penalty assessed 
in each year of the model. It is understandable, but not very 
controllable within the model, that significant penalties are 
assessed in the first two model years. Their magnitude somewhat 
conceals the breakdown of penalties in the later years of the model 
which are of greater interest in the optimization process. TH 
order to focus on the later penalties and reveal the information, 


the analyst can use the hypertext technique to zoom in on the years 
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of interest. By hiding 1989 and 1990 a new graph, Figure 13, is 
created. 

In a Similar way, at any time the analyst may see fit to focus 
on segments of the data that may be of significant concern or 
interest, for example a segment of years where goals are not met, 
a particular role of helicopters, etc. The graphical hierarchy and 
use of a hypertext technique allow the analyst to move about within 
the structure and zoom in on a selected period of time or other 
factor: 

At this stage if the penalties appear to be extreme, the 
analyst may again question the fidelity of the model. A return to 
the data and problem definition may be warranted to assess the need 
for making changes in the penalty structure. In this case, 
however, the penalties do not appear extreme and the analysis 
process continues. 

The analyst sees in Figure 13 a surge of high-technology 
penalties in the middle years of the model, tapering off to almost 
nothing. Mission requirement penalties increase significantly 
beginning in 1997 then decrease somewhat, and there is a steady rise 
in age penalties in the later years. 

Since the requirements for high-technology, average age and 


mission requirements are different for each of the four aircraft 


42 


roles -- attack, scout, cargo and utility -- it is important to view 
the penalties as they effect each role. This reveals the specific 
areas that need further investigation. Figure 13 shows that any 
real trends in penalties will be found in years 1997 to 2008. The 
analyst again focuses on these years in the three appropriate 
penalty areas, Figure 14. 

It appears that Scout aircraft are primarily responsible for 
both the mission requirements penalties, Graph 14a, and the high- 
technology penalties, Graph 14b, especially from 1998 to 2004. In 
later years, Cargo aircraft are responsible for most of the 
remaining penalties in the categories of mission requirements and 
age. 

It is now necessary to look in even greater detail at each 
penalty category and to assimilate information from different parts 
of the model. The windows environment facilitates this part of 
the process by allowing multiple visualization tools to be displayed 
on the screen simultaneously. For example, in considering the 
mission requirement penalties, it is instructive to simultaneously 
view the force composition graphs and the penalty information as in 
Figure 15. 

The analyst is particularly interested at this juncture in the 


scout and Cargo helicopters. In the years where penalties were 
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assessed, the Scout force is composed mainly of OH-58D’s, with the 
OH-58A's and OH-58C's having been SLEPed into 58D's. Producto 
LHX-SCT's begins in the model in the year 2002, at which point the 
build-up returns the Scout force to its minimum requirements and 
penalties are no longer being assessed in this category. This is 
all clearly visible to the analyst from Figure 15. 

It is important for the analyst to examine the reasons that the 
optimization chose this particular course of action and to consider 
such questions as, "Why was the LHX-SCT not procured sooner to 
avoid these penalties?" and "Is it due to production limits, budget 
limitations, or was it driven by O&M costs in the objective 
[oum eo First, however the analysis will continue with an 
examination of the other force composition penalties at this same 
level. 

The force composition of Cargo aircraft from 1997 COS 
primarily CH-47D's. There is no indication of production, and the 
model continues to accrue penalties in force composition for the 
duration of the model. Again, "Why are these penalties incurred and 
what, if any, alternatives are Esas Beginning with 
production limitations as a possible explanation, the analyst 
examines the graph of production line schedules and observes from 


Figure 16 that the production capability for Cargo helicopters must 
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close by 1994. It would not be cost effective to begin production 
earlier since the force structure of Cargo helicopters is adequate 
through 2003 and there is no production capability after 1994. This 
answers one chain of the single-run analysis and could be noted 
for additional exploration. 

Returning to Figure 14, the analyst proceeds with an 
examination of the high-technology penalties which are due almost 
Smet ely 9bO SCOUL -alrcratlL, beginning in 1999 and tapering off 
dramatically at the end of the model. By examining the force 
structure broken down by role in Figure 15, the analyst can see that 
mMNMEIheée exisStypnog 5cout' arrcraft switch from high to low technology 
Bus 999. The Force Composition graph indicates that Scout force is 
almost entirely made up of OH-58D's. The use of hypertext would 
allow immediate access to the information behind the graph. The 
analyst would be able to view and assimilate all this information 
quickly and easily by retrieving these graphs onto the screen and 
Creating an environment like Figure 17. 

Once LHX production begins in 2002, the model's minimum 
fraction high-technology limit is almost attained for Scout 
helicopters by the final year of the model. The analyst may wonder 
what measures would have avoided the penalties. Another look at the 


production schedule and associated spreadsheet information shows 
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that LHX production could begin as early as 1995. Figure 18 shows 
Attack LHX helicopters were procured as early as 1997 and the 
production line was capacitated for the first two production years. 
Note that the force composition graphs do not show LHX Attack 
helicopters in the fleet until 1999. This is due to the lag years 
between purchasing and delivery. Clearly it is more cost-effective 
to procure Attack aircraft and pay penalties for shortfalls in the 
Scout fleet than to procure Scouts. 

Investigation of the age penalties in Figure 14 shows the 
analyst that all penalties are due to Cargo aircraft, beginning in 
year 2004 and escalating to the end of the model. A simultaneous 
look at the production schedule, the maximum age and force 
composition graphs, Figure 19, shows that there is no production 
capability for Cargo helicopters, H-47's, in the later half of the 

The analysis process is controlled by the user and the specific 
approach will depend on the results of the model and the questions 
to be answered. All of the input data and the results are available 
in the spreadsheets and graphs. The way the analyst chooses to 
navigate through this information is subjective. Any trainon 
thought or process of inquiry can be followed. Once the analyst has 


fully explored a single run, it is then possible to begin 


46 


sensitivity analysis by comparing the solution with one or more 


Subsequent runs of the model. 


D. MULTIPLE RUN ANALYSIS 

A single optimal solution is seldom acceptable for a complex 
optimization problem. No solution can account for all possible 
contingencies. As many contingencies as possible need to be 
considered within the realm of analysis. The developers of PHOENIX 
recognized this and included as one of their findings, "Mixed 
integer linear programming provides no useful dual information for 
conducting postoptimality analysis. Sensitivity analysis of model 
output must be accomplished using multiple runs with varying input 
data." (Force Systems Directorate, 1988). 

In the final report on the PHOENIX model (Force Systems 
Directorate, 1988), two solutions were provided, each solved by the 
same model with the same parameters except for the budget growth 
rate. Fourteen other runs were made, but the results were not 
provided (p. 3-10). Since the goal of the study was "to formulate 
and implement a prototype decision aid for force planners for 
evaluating the effect of aviation modernization policy over an 
extended planning horizon" (p. 1-1), the report did not emphasize 
the solutions of PHOENIX but rather its role as a decision aid. The 


application of analysis and visualization principles presented here 
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would have greatly enhanced the ability of the modelers to analyze 
and report on these solutions. Additionally, current users of this 
model would better conduct sensitivity analysis by making the 
multiple runs and evaluating and comparing their results using this 
modeling strategy. 

This section will discuss two ways to perform multiple run 
analysis and apply each to the PHOENIX model. The first scenario, 
which is probably the more common, solves a baseline model with the 
best-known parameters. The analyst may want to explore how 
relaxations or changes to these parameters effect the solution. 
This is typical in many optimization situations, especially in ones 
like PHOENIX that are time sequence problems with multiple measures 
of effectiveness in the objective function and constraints. 

The second approach also involves the changing of parameters, 
however it does not presume that one set best represents the 
solution. Rather it may compare a variety of solutions obtained by 
systematically changing one or several parameters, such as inflation 
rate or budget growth rate. Although this approach was not 
originally reported in the PHOENIX study, it is practical and 
useful, especially when considering a long term planning process in 


an era of ever-changing government spending priorities. Many of the 
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same graphical techniques can be applied to both cases although they 
may be more instructive for one purpose over another. 
ps General Comparisons 

One of the most important aspects of comparisons of 
solutions is the comparison and documentation of the parameters that 
change. Since the input data for each run of the model is displayed 
in graphs or tables, it is relatively simple to produce side-by- 
side comparisons of these values. Highlighting the information that 
is different is also accomplished very easily in these environments. 
Figure 21 demonstrates, for a two case scenario, how effective the 
side-by-side comparison can be in both graphical and tabular form. 
Figure 21a shows the different objective function values and their 
breakdown into O&M costs and penalties. The relative contributions 
of each of these components as well as the actual objective function 
values is quite apparent. Figure 21b, on the other hand, shows the 
change in the purchasing cost of LHX aircraft, a change that can 
drastically alter the outcome of the solution, displayed in 
Spreadsheet form. 

2. Comparison to Base Solution 

The objective function value is typically one of the most 

important criteria for comparison of solutions. However, this is 


not always the best criteria. In a model like PHOENIX which has 
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elastic constraints and associated penalties that are incorporated 
as part of the objective function value, careful consideration must 
also be given to the rest of the solution. It is important to note 
which goals have been met and which have not been met in each 
solution over the time period of the model. It is also important 
to look at the differences between each of the models for each of 
the constraints. For example, Figure 22 shows a comparison of the 
total annual expenditures of two runs of the model. The line graph, 
Figure 22a, indicates the actual expenditures in the two cases but 
Figure 22b is more enlightening as it is a graph of the differences 
in the two solutions. It clearly shows that the expenditures of the 
second run are consistently lower than those of the base case. 
These same types of graphs can be constructed for force 
compositions, procurements and the other constraint conda EEEE 
They can also be used for determining the differences in the 
penalties that are assessed in each situation. 

Another type of graph that can be used to compare results 
of two runs is the scatterplot depicted in Figure 23, which displays 
the same information as Figure 22a. is any deviations from 
the identity, or x-y, line indicate differences in the two 
solutions. A majority of the points to the right of or below the 


x-y line, as in Figure 23, indicates larger values for the solution 
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associated with the horizontal axis, in this case for the base 
eolution. If points were scattered on either side of the line, 
neither solution would be generally better or worse. 

When comparing several different solutions against a base 
case by display in a small multiples format, these graphs become 
quite dramatic and informative. The eye is able to quickly make 
comparisons and 
inadequate solutions may sometimes be discarded. 

3. Comparison of Several Solutions 

The use of multiples described above also works very well 
in the case of the comparison of several solutions. This is 
demonstrated in Figures 24 and 25. Figure 24 is a multiple display 
of the objective function values, broken into O&M costs and 
penalties. The eye can focus on the differences in the data and 
the mind is able to quickly assimilate this information. It is much 
more effective than having to turn pages or even look at two pages 
Simultaneously. Figure 25 is an even more striking display of the 
objective function values. While it does not give quite as much 
information as Figure 24, the boldness of the display and the close 
proximity of the graphs to each other makes assessments of their 


relative values very apparent. 
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Other data analysis techniques can also be used to compare 
several solutions. The use of color serves very well in 
distinguishing between several solutions. A single color associated 
with each particular solution lends consistency to the design and 
can serve to highlight the best solution or at least narrow down the 
field to a few solutions that can be further analyzed. Even when 
color is not available, the use of different shades of gray or 


different textures serve the same purpose, as shown in the PHOENIX 


application. 
When dealing with many possible solutions, it is 
instructive to use some statistical tools. For example, when 


discussing the force composition of attack aircraft in each year of 
the model, it may be useful to consider the minimum, maximum and 
average number of aircraft for each year from all the runs of the 
model. This could be done quite efficiently with boxplots, Figure 
26. These plots can indicate the distribution of the data and show 
any skewing. These boxplots could be used to analyze constraints. 
If a display of the results of multiple runs of the PHOENIX model, 
for example the number of attack aircraft of a certain type procured 
in each year, showed little variation, this could indicate to the 
analyst that this constraint might be tightened or restricted in 


some way without effecting the rest of the model. 
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E. PRINCIPLES APPLIED TO PHOENIX 

The graphs and spreadsheets of the previous sections illustrate 
the application of particular analysis and visualization principles 
to the PHOENIX model. There are, however, some general principles 
enat apply to “all the graphs. 

One of the goals of visualization is to reveal the information 
contained in the data. The eye should focus on the information of 
interest. The graphs are created to bring any outlying values, 
major discrepancies or trends into the forefront. 

Revealing constraint violations is also important. This is 
done by combining bar and line charts. The actual values that the 
solution produces, such as the force composition of attack aircraft, 
are of primary interest, but the analyst also needs to see how this 
number compares with its goal. One piece of information complements 
the other but does not overpower it as it can easily be ignored if 
not of immediate interest. 

Some information is duplicated in a single graph. This is the 
case in Figure 11, the annual expenditures. Both the height of the 
bar and the text at the top of the bar contain the same data. This 
was done because of both the magnitude of the information and its 


importance to the overall solution. 


23 


Another valuable aspect of these graphs, and one of the 
visualization principles, is consistency. Graphs of comparable 
values are consistent in size, coloring and labeling. The order of 
aircraft, both individually and by role, also stays the same. This 
consistency is important to the analyst because it reduces the 
amount of time spent interpreting the data in favor of time spent 
analyzing it. It also facilitates combining similar information, 
such as the number of Scout and Attack aircraft procured in a year, 
which lends a different view to the analysis process. 

Finally, the graphs are designed to highlight and explore the 
relationships among the data values. This is accomplished through 
the windowing environment which allows the overlaying of one graph 
upon or next to another. The relationships can also be explored by 
the display of data values themselves. These values are always 
available in easily accessible spreadsheets. The hypertext 
capability will ensure these are available at the click of a button. 

There are also other features not currently included in this 
analysis due to limited technology but which could be developed to 
be of great assistance. One technique 1S to capture the thought 
processes of the analyst in a scripted scenario that could later be 
reviewed. This could be of significance when a model is only used 


periodically. If the analyst decides to update or review the plan, 
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it would be invaluable to be able to see the process by which 
analysis was initially conducted and how the decisions were made 
when the model was originally run. In many circumstances a new 
analyst might be updating the plan. That analyst could follow the 
scripted version of the initial run and gain much insight and 
understanding prior to conducting follow-up runs and making changes. 

This scripted version could be designed to lead the analyst 
through the input data and results in the same order as was 
Originally done. Voice narration could even be included to further 
document the decisions. The graphs could appear in a flip-chart 
format; they could fade in and out or they could be programmed to 
open and close as appropriate. This script could also be used by 


the analyst to present decisions or options to planners. 
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VII. THE FUTURE OF PHOENIX 


When it was first implemented in early 1988 PHOENIX provided 
the Army with a detailed plan for the Army Aviation Modernization 
Program (AAMP. This plan was revealed in a press release by the 
Secretary of the Army dated September 30, 1988 (Brown et al, 1991) 
which stated: 

The funding provides for an efficient, cost-effective 
production rate of...aircraft in quantities required by the 
Army’s force structure in meeting the requirements of the 
unified and specified commanders-in-chief, and to achieve an 
optimum program within the funding constraints. 
In essence, PHOENIX has guaranteed a modern fleet of helicopters 
within budget that will carry the Army through the next 25 years and 
beyond. Since that time, PHOENIX has also been successfully 
modified and adopted for use in the force planning of tactical 
wheeled vehicles. Additionally, PHOENIX has been expanded to 
include production of component parts and subassemblies of major 
systems as well as transfers from one force package to another. A 
force package is a collection of units grouped together based on 


deployment schedules such as active versus reserve forces. 


(Coblentz, 1991) 
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In addition to PHOENIX, CAA uses another force planning tool, 
the Force Modernization Analyzer (FOMOA). It is a scaled-down non- 
integer linear program version of PHOENIX. It is implemented in 
spreadsheet form on a Macintosh personal computer using the Super 
MacVino optimization package. It is designed for use as a quick 
reaction analysis tool and has a turnaround time of approximately 
twenty minutes. 

These two force planning tools are used to complement one 
another. PHOENIX is the far more capable model but it requires 
well-trained analysts. FOMOA does not require any mathematical 
programming skills but does require user input of such decisions as 
which production lines will open. The constraints of FOMOA are a 
subset of the constraints in PHOENIX and cannot be violated. The 
combination of time, data resources and level of detail required 
dictate which model is used in each circumstance. (Coblentz, 1991) 

PHOENIX requires significantly more input data and its results 
require more extensive analysis than FOMOA. This is one of the 
inherent differences between the two models and cannot be changed 
without changing their purposes. One of the primary advantages of 
FOMOA is its quick turnaround time. PHOENIX originally required 
approximately 10-12 minutes of computer processing (CPU) time for 


a single run, but now runs in one to two minutes. However, CAA 


m 


does not currently have the computer capability to support a run of 
PHOENIX so it must be run on a computer at the Pentagon. This adds 
considerable time to the run of the model, not only physical commute 
time, but also time spent waiting for processing by the Pentagon 
computer. 

The scheduled addition of a workstation to CAA headquarters 
will give them the computer capability to solve the PHOENIX model. 
While it takes more CPU time on a workstation than on a mainframe, 
the total time required to conduct a run of PHOENIX should decrease 
as it would no longer have to compete with other Pentagon projects 
for processing and the commute will be eliminated. The addition of 
an in-house workstation will reduce the time to solve PHOENIX from 
days to minutes. 

Application of the analysis and visualization principles to the 
PHOENIX model as demonstrated in the previous chapter would 
Similarly reduce the time required to conduct a full run of the 
PHOENIX model, from data validation to multiple run analysis. After 
the results of a solution run are entered into the spreadsheet, all 
the worksheets and graphs described in the previous chapter are 
constructed in a matter of seconds. 

This combination of a workstation environment and the modeling 


strategy would bring some of the convenience of FOMOA to the 
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sophistication of PHOENIX. Problems that require the details of the 
PHOENIX model would be solved faster and more efficiently. TE 
would allow more thorough analysis of multiple runs in a shorter 
time and would allow more runs to be made in order to explore a 


greater number of alternate optimal solutions. 
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VIII. CONCLUSIONS 


The widespread availability and technological advances in 
computer capability and the power of information in today’s society 
Support the steady growth of mathematical programming and 
optimization in business, industry, government, and academia. 
Today's real, complex and important problems can be solved through 
the use of large-scale optimization with a fidelity and accuracy 
that was impossible a few years ago. The management and economic 
impact of using operations research is now widely recognized and the 
tools and techniques are being implemented extensively. As bigger 
problems are modeled, the voluminous input and output increase the 
need for effective analysis while complicating its execution, 
frequently making the time and abilities of the analyst the limiting 
factor in the process. The ability to make changes, to update and 
explore the possibilities of the model, and to communicate the 
results to the people who have the problem is time consuming and 
ditirenit: DUE crucial to effective analysis. Thren 
implementation of the modeling strategy developed in this thesis, 
these obstacles can be overcome and thorough, effective validation 


and analysis of large-scale optimizations can become the norm. 
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The principles of analysis and visualization outlined as the 
basis of this modeling strategy are tailored to a specific problem 
to produce the hierarchical structure of customized tools, such as 
graphs and spreadsheets. The analyst controls the process and 
focuses on areas of interest. The application of appropriate 
displays, simple graphical techniques, representation driven graphs, 
and consistency between representations allows the analyst to more 
quickly perceive the extensive information and to readily identify 
trends, outliers, constraint violations, and penalties accrued on 
elastic constraints. The analyst can then relax constraints, change 
penalties, and explore a variety of feasible and alternate optimal 
solutions. The access to source data and relationships among values 
and the backward compatibility of this modeling strategy facilitate 
this analysis. 

Having developed the structure of validation and analysis tools 
for the particular problem, data from subsequent runs of the 
optimization can be ported into spreadsheets and the same analysis 
tools created for this data in a matter of seconds. Multiple 
representations can add new views or information to extend the 
model. Sensitivity analysis and comparisons of multiple runs are 
significantly enhanced by this strategy and by the use of side- 


by-side comparisons, base case comparisons, and multiples. 
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This modeling strategy is effectively applied to large-scale 
optimizations in general but is especially expedient for multi- 
year and recurring models and the complex and subjective problems 
more frequently modeled today. The benefits of creating the tools 
and implementing the strategy are quickly realized. The developed 
model is easily learned by second-generation and subsequent users 
and is easily reviewed between uses. 

As large-scale optimization models play an increasing role in 
industry and government, analysts will seek better ways to represent 
and solve complex problems. In cases where an analyst would 
otherwise not be able to thoroughly assess voluminous data and a 
large array of alternate solutions, implementation of this strategy 
will support thorough analysis and selection of the best possible 


decision: 
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APPENDIX 


INTERACTIVE OPTIMIZATION PROCESS 


ANALYSIS 


ANALYSIS 


ANALYSIS 


ENUMERATION/ MODEL OPTIMIZATION 


VERIFICATION 
CONSOUDATIO 


VALIDATION /FIDELITY 





Figure 1. Interactive Optimization Process. The traditional 
optimization process is indicated by blocks and forward arrows. The 
modeling strategy for the interactive optimization process 
emphasizes the analysis, veriilication/consoLlidation, 
validation/fidelity and multiple run arcs. 
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Figure 2. Budget Constraints. Verification of input values is more 
readily conducted through the use of presentation graphics. As 
expected, the maximum budget input in this case is Steadily 
increasing and the minimum budget input levels off after 
Stabilization of the model. This type of graph supports the 
visualization principles os Simplicity and appropriate displays. 
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Figure 3. Parameter Growth Rates. Input parameters are easily 
compared using a simple bar graph. The visualization principle of 
Simplicity is highlighted here. 
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Figure 4. Policy Parameter Input by Helicopter Role. Minimums and 
maximums should appear as increasing or decreasing functions. The 
drop in Figure 4(b) in 1999 alerts the analyst to a possible input 
error for Attack helicopters. The visualization principle of 
consistency is portrayed by the use of identical line types for type 
of helicopter in each graph, facilitating interpretation. 
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Production Line Schedules 








Possible Closing Years 





Figure 5. Schedule of Production Lines. This graph is useful in 
Biiemvelidation of production line input “data. It also will be 
useful in the analysis of the model to find causes of penalties and 
e Composition shortfalls. It demonstrates the visualization 
principle of data access. 
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Figure 6. Segment of Spreadsheet. Verification of sparse data and 
isolated values is easily conducted in the spreadsheet environment. 
This supports the visualization principles of Simplicity Wawa 
appropriate displays. 
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Figure 7. Total Objective Function Value and Components. This 
graph shows the beginning of the analysis principle of hierarchical 
Structure. The objective function value iS comprised of the O&M 
costs and the penalties. 


69 





‘Annual Objective Function Values 


& 
1200 4 c we anne ee ee | Se. 


NR 








| | | 
Ry « "UM e S 
-. AW : pm E c ST, d T QUE Tu y d 3 S 
1989 > 1992 10D 1905 217737. «1998 3 4.572001 RS 2004 LES 2007 
go A x Le phe ^ et f ; Se : s $ c 
. e . F ' 2i y 51 seg en AS o Y evr, 6 NN Š ^ E: E vx JN 
e : i P < ^ "A n ^ ders < ea”. > ud ous s , 
"ex ^ 7 ne z 4 y xe. s © VE 
^ 7 ae + e ome 1.0, La Me ROOM Se v 
xa A wv EE 6 GEES EN oA 
"NI | nual total Penalties 3 Fcc Coa 
E E LUE cu 4 > ig i is se E VG SA > s rere A easy o, 
2, . e ? a e A ae annie ex F ae ; es c : 
: $ PU » 


pr MATTOS OF DONE 


Figure 8. Annual Objective Function Values and Components. By 
depicting annual components of the objective function, this graph 
is a different level within the hierarchical structuur T HE 
portrays the visualization principle of representation driven 
graphs. 
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Force Violated 
Composition Procurement Constraints 


Figure 9. Hierarchical Structure of PHOENIX. 


What did we NOT get? 





Shortfalls 


A- hierarchical 


Structure 1S common to the modeling strategy for large-scale 
optimizations. The specific elements of the Structure would vary 


bye Specific application. 
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Figure 10. Budget Constraint Violations. The simplicity of the 
line graph makes it easy to see where the optimal annual budget 
violated budget minimums or maximums. This graph also pursues the 
budget branch of the hierarchical structure and represents the 
visualization principle of representation driven graphs. 
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Figure 11. Annual Expenditures by Component. This graph continues 
in the hierarchical structure and portrays the visualization 


principles of representation driven graphs, appropriate displays, 
and simplicity. 
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Figure 12. Total Annual Penalties. Analysis within the 
hierarchical structure can proceed on a different branch, as 
demonstrated by switching from budget data to penalty data. 
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Figure 13. Penalties components for selected years. The use of the 
zooming principle of visualization allows the analyst to view in 
greater detail the penalty values for selected years of interest. 
This graph also highlights the principles of hierarchical structure, 
consistency and appropriate displays. 
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Figure 14. Type of Penalty Incurred by Aircraft Role. The 
hierarchical structure aids in the identification of specific 
penalties incurred and their relative amounts. The use of the 


zooming technique focuses the portrayal of only those years where 
penalty values were significant to the model. This set of graphs 
also demonstrates consistency, simplicity and appropriate displays. 
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Figure 15. Application of Windowing Environment. This allows for 
efficient on-screen comparisons of additional views of information 
within the hierarchy. The combination of bar and line charts 
(front) helps the analyst quickly spot violations of constraints. 
This is an example of a side-by-side comparison. 
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Figure 16. Application of Windowing Environment. This view 
supplements the on-screen comparisons in Figure 15 with the 
Production Line Schedules shown in Figure 5 to enable the analyst 
to identify potential causes of the Mission Requirement constraint 
violations for specific aircraft models. The side-by-side 
comparisons and the user controlled environment are key principles 
at this stage of the analysis. 
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Figure 17. Windowing Environment with Spreadsheet. The user 
control and data access principles make it easy for the analyst to 
view the figures in the appropriate spreadsheet that support an 
aspect of a graph under investigation. The windowing environment 
supports the side-by-side comparison of this information. 
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Figure 18. Windowing Environment for Side-By-Side Comparisons. The 
on-screen combination of a spreadsheet and two types of graphs 
demonstrates analytical advantages. It highlights the principles 
of user control, data access and appropriate displays. 
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Figure 19. Continued Application of Windowing Environment. 
Approaching the lowest level of one branch of the hierarchical 
Structure, the analyst can use the side-by-side comparisons to 
answer questions about the adequacy of the solutions. 
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Figure 20. Aircraft Procurement Data for Combined Roles. This 
graph illustrates the capability to collect data from four roles 
into two combinations. The analysis principle of extensibility 
allows for this user controlled addition to the analytical tools and 
backward compatibility would ensure that this data change would be 


filtered throughout the graphs and spreadsheets in all previous 
runs. 


82 


on Pir m: pe t "5 te wm ont sss x 
* vi 


T otal Objective Function Base Case . | | Total Objective Function Solution °° 
ó ! eon e id e 





ay 





E D 


38888383 





rrr ATION OF WOMAS 77 7777 





I Total O&M sh Tova Penaltos | m. 


»u De ¿Xy e T ws OVE: 0, SPUREN n. i A 


Tota ios Function Value 
JB ga ^i b 





oon 
ia 

















Figure 2la. Two Case Comparison of Objective Function Values. 
Multiple run analysis is enhanced by the ability to create side- 
by-side comparisons of subsequent runs to the same information from 
a base case. 
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Figure 21b. Two Case Comparison of Purchase Cost. Multiple run 
comparisons can be conducted in the spreadsheet environment as well 
as graphically. This supports the analysis principles of base case 
and side-by-side comparisons and, in conjunction with figure 21a, 
that of multiple representations. 
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22(b) 
Figure 22. Multiple Run Comparisons. In figure 22a a simple line 
graph serves to compare total budget expenditures for a base case 
to an alternate solution. Figure 22b is a difference bar chart 
supported by a formula that subtracts the expenditures for the 
alternate solution from those of the base case. 
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Figure 23. Scatter Plot Comparison. In this comparison of a base 
case to an alternate solution deviations from the identity (x=y) 
line are readily apparent. A majority of the points to the right 
of and below the line indicate larger values for the base solution 
associated with the horizontal axis. 
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Figure 24. Objective Function Values for Multiple Solutions. The 
use of multiples facilitates the display and comparison of a variet 


æ er we 


of solutions with differences and similarities readily apparent. 
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Figure 25. Multiples Displayed with Area Graphs. The use of area 
graphs presents the objective function values in a more dramatic 
visualization with the same focus on similarities and differences. 
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Figure 26. Boxplot. Statistical tools, such as the boxplot, can 
be used to compare results of several runs of the model. Results 
can be used to tighten or restrict constraints without effecting the 
rest of the model. The horizontal line segment inside each box is 
the 50th percentile and the top and the bottom of the box indicate 
the 75th and 25th percentiles. The ends of the vertical lines are 
the adjacent values which represent the largest or smallest 
observations within 1.5 times the difference between the 75th and 
25th percentile on either end of the box. The dots above or below 
these lines represent outlier values. 
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